A detailed discussion of background information is set forth in parent applications U.S. patent application Ser. Nos. 10/822,445 and 11/034,325 and is incorporated by reference herein. Some more pertinent background is set forth below. All of the patents, patent applications, technical papers and other references referenced below and in the parent applications are incorporated herein by reference in their entirety. Various patents, patent applications, patent publications and other published documents are discussed below as background of the invention. No admission is made that any or all of these references are prior art and indeed, it is contemplated that they may not be available as prior art when interpreting 35 U.S.C. § 102 in consideration of the claims of the present application.
1. Communication with Other Vehicles
The RtZF™ system of this invention can incorporate vehicle-to-vehicle communication allowing vehicles to inform other vehicles of their location, velocity, mass etc.
U.S. Pat. No. 5,506,584 to Boles relates to a system for communication between vehicles through a transmit and transponder relationship. The patent mentions that there may be as many as 90 vehicles within one half mile of an interrogation device in a multi-lane environment, where many of them may be at the same or nearly the same range. Boles utilizes a transponder device, the coded responses which are randomized in time, and an interrogation device which processes the return signals to provide vehicle identification, speed, location and transponder status information on vehicles to an operator or for storage in memory. No mention is made of how a vehicle knows its location or how accurate that knowledge is and therefore how it can transmit that location to other vehicles.
U.S. Pat. No. 5,128,669 to Dabbs provides for two-way communication and addressing messages to specific vehicles. This is unnecessary and the communications can be general since the amount of information that is unique to one vehicle is small. A method of handing bi-directional communication is discussed in U.S. Pat. No. 5,506,584 to Boles. A preferred vehicle-to-vehicle communication system using pseudonoise techniques is more thoroughly discussed below.
In embodiments of the invention described herein, vehicle-to-vehicle communication is used, among other purposes, to allow the fact that one vehicle knows its position more accurately than another to use communication to cause the other vehicle to also improve the accuracy with which it knows its position.
2. Infrastructure-to-Vehicle Communication
2.1 General
The RtZF™ system of this invention can also incorporate communication between a vehicle and infrastructure for a variety of reasons including obtaining the latest map updates, weather conditions, road conditions, speed limits, sign contents, accidents ahead, congestion ahead, construction, general Internet access prior to the time that there is a ubiquitous broadband network in place that is accessible from a moving vehicle, and for many other reasons.
The DGPS correction information can be broadcast over the radio data system (RDS) via FM transmitters for land use. A company called Differential Correction, Inc. has come up with a technique to transmit this DGPS information on the RDS channel. This technique has been used in Europe since 1994 and, in particular, Sweden has launched a nationwide DPGS service via the RDS (see, Sjoberg, Lars, “A ‘1 Meter’ Satellite Based Navigation Solutions for the Mobile Environment That Already Are Available Throughout Europe”). This system has the potential of providing accuracies on the premium service of between about 1 and 2 meters. A 1 meter accuracy, coupled with the carrier phase system to be described below, provides an accuracy substantially better than about 1 meter as preferred in the Road to Zero Fatalities™ (RtZF™) system of this invention.
In addition to the FM RDS system, the following other systems can be used to broadcast DGPS correction data: cellular mobile phones, satellite mobile phones, satellite Internet, WiFi, WiMAX, WiMobil MCA (multi-channel access), wireless tele-terminals, DARCs/RBDS (radio data systems/radio broadcast data system), type FM sub-carrier, exclusive wireless, and pagers. In particular, DARC type is used for vehicle information and communication systems so that its hardware can be shared. Alternately, the cellular phone system, coupled with the Internet, could be used for transmitting corrections (see, Ito, Toru and Nishiguchi, Hiroshi entitled “Development of DGPS using FM Sub-Carrier For ITS”). Primarily, as discussed elsewhere, vehicle-to-vehicle communications can be used to transmit DGPS corrections from one vehicle to another whether the source is a central DGPS system or one based on PPS or other system.
One approach for the cellular system is to use the GSM mobile telephone system, which is the Europe-wide standard. This can be used for transmitting DGPS and possibly map update information (see, Hob, A., Ilg, J. and Hampel, A. entitled “Integration Potential Of Traffic Telematics).
In Choi, Jong and Kim, Hoi, “An Interim Report: Building A Wireless Internet-Based Traveler's Information System As A Replacement Of Car Navigation Systems”, a system of showing congestion at intersections is broadcast to the vehicle through the Internet. The use of satellites is discussed as well as VCS system.
This is another example of the use of the Internet to provide highway users with up-to-date traffic congestion information. Nowhere in this example, however, is the Internet used to transmit map information. In fact, once there is an Internet or equivalent connection to a vehicle then other information can be transmitted such as updated map information, weather and visibility, local conditions ahead, accident information, congestion information, DGPS corrections, etc. In fact, with a high bandwidth Internet connection, much of the computations, especially safety related computations, can best be done on the Internet where the system reliability would exceed that of a vehicle-based system. The forecast that “the network is the computer” (as prompted by Cisco Inc.) will begin to become reality. The crash of a safety related processor due to a software bug could not be tolerated in a safety related system and would be less likely to occur if the critical computations occur on the network. Furthermore, upgrades to vehicle-based software also become feasible over such a high bandwidth connection.
A paper by Sheu, Dennis, Liaw, Jeff and Oshizawa, Al, entitled “A Communication System For In-Vehicle Navigation System” provides another description of the use of the Internet for real traffic information. However, the author (unnecessarily) complicates matters by using push technology which isn't absolutely necessary and with the belief that the Internet connection to a particular vehicle to allow all vehicles to communicate, would have to be stopped which, of course, is not the case. For example, consider the @home network where everyone on the network is connected all the time.
A paper by Rick Schuman entitled “Progress Towards Implementing Interoperable DSRC Systems In North America” describes the standards for dedicated short-range communications (DSRC). DSRC could be used for inter-vehicle communications, however, its range according to the ITS proposal to the Federal Government would be limited to about 90 meters although there have been recent proposals to extend this to about 1000 meters. Also, there may be a problem with interference from toll collection systems, etc. According to this reference, however, “it is likely that any widespread deployment of intersection collision avoidance or automated highways would utilize DSRC”. Ultra wide band communication systems, on the other hand, are a viable alternative to DSRC as explained below. The DSRC physical layer uses microwaves in the 902 to 928 megahertz band. However, ITS America submitted a petition to the FCC seeking to use the 5.85 to 5.925 gigahertz band for DSRC applications.
A version of CDPD, which is a commercially available mobile, wireless data network operated in the packet-switching mode, extends Internet protocol capabilities to cellular channels. This is reported on in a paper entitled “Intelligent Transportation Systems (ITS) Opportunity”.
According to a paper by Kelly, Robert, Povich, Doublas and Poole, Katherine entitled “Petition of Intelligent Transportation Society of America for Amendment Of The Commission's Rules to Add Intelligent Transportation Services (ITS) As A New Mobile Service With Co-Primary Status In The 5.850 to 5.925 GHz”, from 1989 to 1993 police received an annual average of over 6.25 million vehicle accident reports. During this same period, the total comprehensive cost to the nation of motor vehicle accidents exceeded the annual average of 400 billion dollars. In 1987 alone, Americans lost over 2 billion hours (approximately 22,800 years) sitting in traffic jams. Each driver in Washington D.C. wastes an average of 70 hours per year idling in traffic. From 1986 to 1996, car travel has increased almost 40% which amounts to about a 3.4% increase per year.
Further, from Kelly et al., the FCC has allocated in Docket 94-124, 46.7 to 46.9 GHz and 76 to 77 GHz bands for unlicensed vehicular collision avoidance radar. The petition for DSRC calls for a range of up to about 50 meters. This would not be sufficient for the RtZF™ system. For example, in the case of a car passing another car at 150 kilometers per hour. Fifty meters amounts to about one second, which would be insufficient time for the passing vehicle to complete the passing and return to the safe lane. Something more in the order of about 500 meters would be more appropriate. This, however, may interfere with other uses of DSRC such as automatic toll taking, etc., thus DSRC may not be the optimum communication system for communication between vehicles. DSRC is expected to operate at a data rate of approximately 600 kbps. DSRC is expected to use channels that are six megahertz wide. It might be possible to allocate one or more of the six megahertz channels to the RtZF™ system.
On DSRC Executive Roundtable—Meeting Summary, Appendix I—Proposed Changes to FCC Regulations covering the proposed changes to the FCC regulations, it is stated that “ . . . DSRCS systems utilize non-voice radio techniques to transfer data over short distances between roadside and mobile units, between mobile units and between portable and mobile units to perform operations related to the improvement of traffic flow, traffic safety and other intelligent transportation service applications.”, etc.
A state or the Federal Government may require in the future that all vehicles have passive transponders such as RFID tags. This could be part of the registration system for the vehicle and, in fact, could even be part of the license plate. This is somewhat discussed in a paper by Shladover, Steven entitled “Cooperative Advanced Vehicle Control and Safety Systems (AVCSS)”. AVCSS sensors will make it easy to detect the presence, location and identity of all other vehicles in their vicinity. Passive radio frequency transponders are discussed. The use of differential GPS with accuracies as good as about two (2) centimeters, coupled with an inertial guidance system, is discussed, as is the ability of vehicles to communicate their locations to other vehicles. It discusses the use of accurate maps, but not of lateral vehicle control using these maps. It is obvious from reading this paper that the author did not contemplate the safety system aspects of using accurate maps and accurate GPS. In fact, the author stresses the importance of cooperation between various government levels and agencies and the private sector in order to make AVCSS feasible. “Automotive suppliers cannot sell infrastructure-dependent systems to their customers until the very large majority of the infrastructure is suitable equipped.”
A related art document for collision avoidance in general is U.S. Pat. No. 6,487,500, GPS Vehicle Collision Avoidance Warning and Control System and Method, to Lemelson et al. (Lemelson). Although this patent addresses many aspects of collision avoidance, in particular, it addresses vehicle-to-infrastructure communication so it will be reviewed in some detail. However, prior to going in to detail, a few general comments are in order.
It is believed that the system described in Lemelson, as best understood by one or more of the inventors of inventions disclosed herein, is unlikely to be functional. It relies on many papers and patents in the prior art but does not explain how any of these prior art ideas would be implemented. For example it does not handle the problem of obstructions or potential collisions when visibility is poor since there is virtually no general illumination provided, especially in the IR portion of the spectrum, and radar is generally not capable of providing images that can be segmented, analyze and identified. Furthermore, without accurate maps, it is unlikely that run-off-the-road situations, stop sign infractions, stoplight infractions or collisions with systems that do not have the Lemelson system and thus cannot communicate their positions and velocities to the host vehicle can be solved. There is also no discussion of an IMU (Inertial Measurement Unit) or gyroscopes being present and if so how errors in these devices would be corrected. The system relies completely on multiple antennas receiving GPS signals which are notoriously poor in urban settings. It also relies on pseudolites which, although not defined in this patent, in the literature contain atomic clocks. In view of multipath delays and uncertainties, a large number of such pseudolites would need to be deployed to cover the entire continental United States. Each of these pseudolites is comparable in costs to a GPS satellite due to the need for an atomic clock. Thus, the deployment of pseudolites is totally impractical. Similarly, use of local area differential GPS corrections is impractical since a system would need to be placed approximately every roughly 30 to 60 miles across the entire United States. Wide area differential GPS, which is not mentioned in the patent, is an appropriate solution since only a few such stations are needed. Although Lemelson speaks of using neural networks to identify objects in an image, this is not easily accomplished unless three-dimensional data is available. Other than a brief mention of stereo cameras, this patent does not describe any method of getting a three-dimensional image. Stereo photography only works when the device being imaged is near to the host vehicle and thus this system cannot be used to segment objects that are more than about ten meters from the vehicle. Thus, three-dimensional information on most objects is not available. Furthermore, obtaining three dimensions from stereo cameras is a computationally intensive activity. Additionally, although Lemelson is concerned with the recognition of objects with which the vehicle is about to collide, it does not mention using this identification as a means for deploying occupant restraint systems within the vehicle, i.e., as an anticipatory crash sensor. In short, Lemelson is not likely to be enabling of any practical GPS vehicle collision avoidance warning and control system and method.
To summarize, Lemelson provides a outline of how a collision avoidance system might operate and provides a plethora of patent and non-patent prior art and leaves it to the reader to sort out what information from this prior art would be applicable to this system. Many of the problems that Lemelson assumes are solved by the prior art in fact are not. For example, prior to the filing date of the '452 provisional patent application, there were not believed to be any viable vehicle-to-vehicle communication systems available, except as provided by Automotive Technologies International, Inc. (ATI) and Intelligent Technologies International Inc. (ITI), nor were there any vehicle-to-infrastructure communication systems believed to be available other than using cell phones. Cell phones do not have sufficiently high bandwidth to permit exchange of information in a collision-imminent situation plus there are many areas of the country where cell phone coverage is not reliable.
If, as described below, a ubiquitous network system such as the Internet is installed with transmitting towers located at every one hundred miles or less, then a somewhat local area differential GPS system could be installed and the differential corrections from each of those local stations could be used to augment and make the wide area differential GPS system more accurate. Atomic clocks would not be installed and thus these stations could not act as pseudolites.
In the Summary of the Invention section, Lemelson notes use of GPS to locate a vehicle on a highway. Since the vehicle does not know where the highway is, except perhaps crudely in good weather, it cannot locate itself on the highway. Although pseudolites signals are mentioned, there is no discussion of how the errors in pseudolites signals are corrected. Such errors are caused by multipath which results whenever the vehicle does not have a line of sight path to the pseudolites. To communicate their position with centimeter accuracy to a center from a plurality of automobiles over the only available network, at the time Lemelson was filed, cell phones, would saturate bandwidth available from such cell phones. Similarly, to receive GPS coordinates of all vehicles that might surround a vehicle when operating on a major freeway in congestion conditions would require that the vehicle communicate perhaps with one thousand or more vehicles which again would saturate or exceed any known communication system at that time. No mention is made of a method of determining which vehicles with which to communicate and how the communication system should decide. Thus, the inter-vehicle communication system described in Lemelson is unworkable and therefore not enabled. Note that for collision avoidance purposes, the communications between potentially colliding vehicles ideally would take place at least once every ten milliseconds and certainly at least once every one hundred milliseconds.
Although Lemelson frequently mentions that when a collision is unavoidable, means would be implemented to minimize any injury or damage therefrom, no description is made as to how this would be accomplished other than avoidance maneuvers using a fuzzy logic vehicle control system. Similarly, mention is frequently made of determining the vehicle's attitude using the output from a multiplicity of antennas which receive the same GPS signals. To calculate an angular velocity of a vehicle from this system is possible and not computationally intensive. However, it is well known that differentiating a signal results in a significant loss in accuracy. Thus, to obtain an angular velocity from the angular position determination is possible but it will have a substantial error. To further differentiate the velocity signal to obtain an angular acceleration results in a situation where the angular acceleration errors exceed the calculated value. To use this system to predict a rollover or skidding condition is possible but error prone to false positives. A preferred approach is to use gyroscopes or preferably an IMU as disclosed herein.
Lemelson further suggests that all areas surrounding the vehicle can be scanned and all potential hazards and objects can be segmented and analyzed and their potential for collision hazard determined. This capability was certainly not in the state-of-the-art in 1993. It is only now becoming feasible to consider such a scheme. Additionally, no mention is made in Lemelson as to how this would be accomplished at night or under other low visibility conditions.
Lemelson mentions that the detection of objects surrounding the vehicle would be conducted using scanning by one or more television cameras. No mention is made as to how this would be accomplished. In using the word scanning, it is unclear from the description whether Lemelson is referring to a raster scan system that is a part of television cameras today or to physically changing a direction of view of the television cameras? Since the physical motion of the television camera would be too slow, one can assume that it is the raster scan system that Lemelson is discussing which would require many cameras to cover all of the areas surrounding a vehicle. No mention is made for example of a scanning laser beam that would illuminate a potential target to be imaged by the television camera. Note that television by itself implies using the visual part of the electromagnetic spectrum which may work on a nice clear day but has serious problems in rain, snow, smoke or fog and at night without illumination.
Radar scanning in also mentioned and in this case since no mention is made of phased array technology, one must assume that the radar scanning is done by moving the radar antenna. Cost-effective solid-state radar phased array antennas were developed long after 1993. Therefore, they could not have been contemplated at the time of Lemelson. Radar scanning by physically moving the antenna is very costly and slow and thus not practical for surround vehicle scanning.
Thus there is no enabling disclosure that would permit the Lemelson system to detect obstacles, people, bicycles, animals, signs, terrain, roadway features and turns or the like.
Lemelson suggests displaying a visually perceptible symbol on a windshield of one automobile including the relative position and motion between the one automobile and any collision threat. Again no mention is made as to how this would be done to make it useful to the driver. Two methods disclosed herein and in the reference patents to ATI and ITI include an icon displayed as seen from above and the projection of icons of the threatening objects onto the windshield in the field of view of the driver such that the icon appears where the object is if the driver could see it. None of this is suggested in Lemelson.
In the Detailed Description of the Preferred Embodiment section, Lemelson states that he uses the same pseudolites to both transmit differential correction information as well as to transmit its own satellite-like beacon. Lemelson also states that the pseudolites tower is in the same general vicinity as the motor vehicles therefore reinforcing the fact that the pseudolite towers transmit local differential GPS corrections and therefore there will need to be many of them.
Although the vehicle in Lemelson does not maintain its own map, it does obtain the location of all hazardous objects from the control center. Needless to say this would be a massive and impractical method of communicating map features to a vehicle for all vehicles in the vicinity. As for communications, Lemelson describes a number of different methods without any indication as to how the vehicle would segment information from potentially threatening vehicles from that of other non-threatening nearby vehicles.
The communication system proposed in Lemelson is difficult to understand. It appears that all vehicles communicate with each other and also with the control tower. If the control tower is fifty miles away then thousands of vehicles can be within a fifty mile radius of the host vehicle resulting in a massive amount of communication which undoubtedly would exceed the available bandwidth for many places in the US. This can be reduced by providing control towers which are closer together however, since each control tower is a pseudolite with an atomic clock, the cost of this implementation would be prohibitive.
One analogy is in order. Although an FM and AM radio tuner can literally receive hundreds of stations it can only effectively listen to one at a time. Presumably, a large hardware installation at a pseudolite could house such hardware to listen to thousands of vehicles simultaneously but the cost of such an installation would be high.
Lemelson does suggest a combination of radar with an imaging system which would presumably give the distance to an object if it had been first segmented in the image. However, this cannot be used to aid in the segmentation process so it is unlikely that such a system will be used. Absent is any discussion of how this would be done in inclement weather or at night or in the presence of other vehicles where the radars would interfere with each other.
One important point is that in no place does Lemelson indicate how the pseudolite obtains an accurate time measure so that it knows when to transmit. In the literature on pseudolites, it is generally assumed that a pseudolite would have an atomic clock. An atomic clock would clearly render the pseudolite prohibitively expensive to be used in the manner in which Lemelson suggests. Alternate methods of obtaining accurate time base might include the use of a less accurate and a less expensive clock coupled with a periodic correction as obtained when solving GPS equations. However, the GPS equations only allow you to determine your location within some number of meters. One would assume therefore that the accuracy by which you could obtain time similarly would be orders of magnitude less accurate than an atomic clock. Although new clocks are being developed which approach atomic clock accuracies at much less expense, these were certainly not available nor contemplated in 1993. Therefore, for a pseudolite to be able to transmit a PNR that is of comparable integrity as from a satellite in 1993 would require the presence of an atomic clock thus rendering this device impractical and thus the entire Lemelson system is not capable of being enabled as described.
Lemelson suggests that an image field analyzing computer using neural networks, artificial intelligence and fuzzy logic can identify objects on the road such as other vehicles, pedestrians, barriers and dividers, turns in the road, signs and symbols and generate identification codes and detect distances from such objects by their size and shape. Lemelson does not say how this would be accomplished nor is it described in prior art literature reference by Lemelson.
Lemelson mentions the possibility of using infrared imaging methods with an intensifying electron gun but does not mention illuminating the scene with infrared. Therefore, only passive infrared can be considered. Similarly, Lemelson mentions stereo imaging without any discussion of illumination.
Lemelson mentions that the distance to an object can be determined since an object has a known width. This assumes that the object has been segmented from other objects and the background. Thus the method of measuring distance to the object cannot be used as a method of segmenting objects within an image. However, the segmentation process will undoubtedly provide, as a byproduct, the distance to the object. Perhaps object size can be a factor in the segmentation process but this is not disclosed or suggested by Lemelson.
In the discussion on image processing, it is stated that the signal is input directly into the neural network processing elements, thus he has not considered the possibility of edge detection algorithms, or other methods, where there would be significant pre-processing before the results are input into a neural network. Also there is no discussion of segmenting the particular elements in the image or objects so that recognition can take place. It would be virtually impossible in 1993 to look at the image containing multiple vehicles and to analyze that whole image in one neural network. The individual objects must first be isolated and segmented before there is the part of image processing leading to recognition. There is no mention of this in Lemelson and once more there is nothing to teach one skilled in the art how to practice the Lemelson invention. In another location, Lemelson mentions that the image data is routed to selected virtual processing elements which implement the neural network computing function. In other words, there has been no edge detection, other pre-processing or segmentation.
Lemelson says that the system can also be responsive to lane markers, safe speed markers, curve warnings or other hazards indicating devices installed along or in the roadway and these can be either visually or electronically detectable. No further description is made and therefore this does not teach someone how to practice the invention. It is uncertain as to what kind of visual road markers or what kinds of electronic road markers Lemelson is referring to? Certainly not signs since nothing has been said about interpreting the shapes or nomenclature that would appear on a sign. Also, no such electronic devices exist on highways today so therefore the question remains to what devices Lemelson is referring to?
In another location, mention is made of the possible existence of maps without any indication as to the accuracy of these maps or from where they are derived. Since it is under the heading of navigational computer, it is probably navigation type maps which have an accuracy on the order of ten to one hundred meters. In another place, mention is made of the possibility of displaying map graphics on a heads-up or video display. No indication is made that this display can be used to direct the driver as to where he should turn or that it should be in his line of sight corresponding with the road if it is on a heads-up display.
Lemelson does indicate the possibility of having radio receivers and systems employing solid-state lasers and photodetectors of reflected laser light to provide coded information that may be placed at appropriate places along the highway. This appears to be for informational purposes only and not for location purposes since there is no description of information that would be transmitted. Thus a highway can be provided with an electronic message or a barcode for example that would be scanned by the system to obtain some such information. This is not an imager since it only uses a photodetector and not an imaging array. In fact, nowhere in Lemelson is mention made of use of an optical scanner in conjunction with the CCD or other array.
At another place, Lemelson mentions monitoring erratic driving on the part of the driver. This is done by measuring such things as the change in vehicle attitude and erratic driving patterns along with uneven unnatural acceleration and similar such activity. However, it fails to compare this with the road that the driver is driving on and thus what might be erratic driving on a smooth highway could be very normal driving on a country road with potholes, for example, in this case the driving would seem to be highly erratic and give many false positives.
2.2 Ubiquitous Broadband Network
What is believed to be the first practical ubiquitous broadband network will be described below. Although many have proposed such a network, until now there has been no enabling reason why it should be adopted. The network can be built on the Wi-Fi 802.11 standards or on a new, yet to be developed standard based, for example, on ultra wideband. Naturally as in cellular networks, one key ingredient is the handoff as the vehicle travels from cell to cell. This problem has been dealt with in a news release “Faster Handoff Between Wi-Fi Networks Promises Near-Seamless 802.11 Roaming”, Apr. 13, 2005.
U.S. Patent Application Publication No. 20040014463 illustrates a poor method of updating a non-vehicle database through the use of data-encoded audio CDs. This method would be obsoleted by the ubiquitous broadband network described herein.
2.3 Electronic Local and Emergency Communication from Infrastructure
Also described below herein is a method for communicating emergency information, such as “bridge out”, “road temporarily closed”, “construction ahead”, “road icy ahead”, etc., to vehicles using a time critical vehicle-to-vehicle communication system. Through this technique, a central station or local transmitters can send such emergency information to all affected vehicles.
2.4 Precise Positioning without GPS
Various methods of a vehicle learning its precise position without the use of GPS are discussed in several patents and patent applications to ITI referenced herein. Some of these methods can take the form of infrastructure-to-vehicle communication or vehicle-to-infrastructure-to-vehicle communication. There appears to be no related art to these concepts that predate their disclosure in the ITI patents and applications.
2.5 DGPS Corrections from Infrastructure
The following is a review of patent U.S. Pat. No. 6,628,233 to Knockeart et al. In certain aspects, some of the inventions herein can be thought of as vehicle information systems with many similarities to the '233 patent. In particular, the '233 patent describes a method for obtaining DGPS corrections from a central server over a cell phone link.
The vehicle information system of the '233 patent comprises an in-vehicle system and a centralized server system. The in-vehicle system communicates with the server system using a wireless communication link. This is similar to some of the disclosure below, one difference being that most of the work is done on the in-vehicle computer in the system herein and the server serves primarily to update various aspects of the database that resides in the vehicle resident computer. Since the in-vehicle system herein should have an accurate map, the server-based system serves primarily to update that map and does so frequently both with temporary and permanent changes. Another difference is that the communication to the server from the system described herein in general can be accomplished over a specialized network designed for that purpose, such as a ubiquitous broadband wireless network that can be, e.g., the Internet. The cell phone system, although used occasionally, would not be the primary link as it is in the '233 patent.
It is important to note that the concept of finding an address or a location based on the phone number is believed to have been first disclosed in applications filed by ITI and ATI. An added feature to that system as described in more detail below involves finding the location of a mobile phone and guiding the vehicle to that location.
The '233 patent describes methods for receiving from a server a position correction to where the driver believes he or she is located. This position correction is on the order of one hundred feet or more and does not correspond to the methods for correcting vehicle position at the centimeter level as described herein.
The '233 patent mentions a roadside optical or radiofrequency beacon system that can be used to provide location information to the vehicle. This concept was also disclosed in prior ATI or ITI patents. The '233 patent also mentions that although the digital or analog cellular system is a preferred link, a satellite-based system can alternately be used. Again these are concepts disclosed in prior ATI or ITI patents.
The '233 patent also mentions the use of GPS correction data to obtain differential accuracy. However this concept was disclosed prior ITI and ATI patents. Various methods are mentioned for obtaining this accuracy starting with the inverted DGPS wherein the vehicle sends its raw readings from the GPS satellites to a server which then incorporates the DGPS corrections and sends it back the actual location. This system would be too slow for the invention herein especially if it took place over the cell phone network. In none of these approaches is the '233 patent talking about sub-meter accuracies. Their problem is that the GPS according to the '233 patent is only accurate to one hundred meters and that is not sufficiently accurate in many cases for the route guidance system of the '233 patent.
It is important to note that the “off road tolerance” is initialized to one hundred feet and grows linearly to a maximum of five hundred feet under the '233 patent. Under the inventions herein, the vehicle driving tolerance is measured in centimeters.
The '233 patent mentions that the in-vehicle system can also compute its own GPS correction data when it knows its location precisely. By precisely is meant that when it makes a turn for example at a maneuver point. This preciseness is measured in terms of many feet not in terms of centimeters. Thus, the use of physical location information to obtain accurate DGPS corrections as disclosed in ATI and ITI patents is not contemplated by the '233 patent.
2.6 Route Guidance
Route guidance can be done on the vehicle and not on the server as in the '233 patent. In the system described herein, route guidance would generally be done based on finding one or more paths from the current GPS location to the GPS location of the destination. The translation of an address or phone number or name of a point of interest to a GPS location can be done either on the vehicle or more likely, at a special server site over the Internet.
It is interesting to note that all of the factors which cause errors in the vehicle driver reaching his destination are eliminated in inventions described herein. These factors are described in the '233 patent and include inaccuracy in the estimate of the vehicles of actual location. This would not happen on systems described herein since the accuracy with which the vehicle knows its location is measured in centimeters instead of hundreds of meters. Another cause is errors in the system's map of the road network. This again cannot happen or use of systems herein would result in accidents.
Another cause is inaccuracy in estimating the distance traveled by the vehicle which again could not happen in systems described herein. Thus, with systems described herein, the route guidance system described in the '233 patent would be substantially improved.
Another key feature of the present invention is to display a map along with a color marked, for example, pathway on a heads-up display thus eliminating the need for the driver to change his gaze either to a small heads-up display or to an alternate vehicle display. Practicing inventions described herein, the driver constantly keep his eyes on the road.
The in-vehicle map system in the '233 patent has only the main roads pre-loaded. It does not include residential roads. A system described herein includes all roads to centimeter accuracy thus, once again, the route guidance system of the '233 patent would be substantially improved using a system described herein.
2.7 Display of Pictures
A further enhancement described in more detail below is to obtain a picture or photograph of a point of interest and to download that photograph to the vehicle over the network connection. This picture can be one that has been obtained earlier and resides in a database or it can be a satellite-derived picture such as obtainable from Google Earth, or a similar application. It also could be a photograph of a traffic jam recently obtained from a satellite or airplane providing the driver with a bird's eye view of the traffic that lies ahead of him or her. This can also be in the form of an icon display.
2.8 In-Vehicle Signage
The '233 patent provides the opportunity for a driver to observe on a display the text of signs that would be at the next maneuver point. This is only available when the vehicle is operating in the route guidance mode and only a limited class of such signs are displayed. In inventions herein, the road signs including speed limits, warning, even temporary message signs as well as navigational signs would all be available on a display for the vehicle operator and also could be available in any language. The vehicle operator would have the ability to scroll forward or backward to see what signs are coming up or what signs the driver may have missed due to the blockage from a truck, for example. Thus, rather than having signs available only at maneuver points and only in the route guidance mode and further only showing a small subset of the signs that are on the road, the inventions herein provide displayed signage at any time and at all times. The signage can be displayed on any kind of a display including an in-vehicle display and in fact, with occupant sensing capability, the sign can be placed on the windshield where the driver would see it if it were not blocked by a truck or obscured by fog.
The article Hamblen, Matt “Wireless Project Delivers Quick ROI for Truckers”, Computerworld Jul. 12, 2005, describes the advantages that sending and receiving of text messages on a PDA or Blackberry has for the trucking industry. Sending and receiving of such messages requires that the truck driver stop his vehicle, thus wasting valuable time. There is a need, therefore, for a message system for short messages where the display can be on a heads-up display in the view of the driver and where the response can be oral with a speech translator for outgoing text messages.
2.9 Network is the Computer
Although a preferred location of the computer that operates the communication and information system that resides herein is in the vehicle itself, this need not be the case and when the ubiquitous broadband network becomes a reality some or all of the computational functions could be done on the network using whatever and wherever there is computational power available. This is similar to the Cisco Corporation idea that “the network is the computer”. In this case, the network can provide computational services just as if the computer were resident on the vehicle. This is different than the '233 patent where the dedicated central server provides the route guidance answers. This is a subtle but important distinction as the computations can be done anywhere on the network and are not limited to a dedicated computer with a substantial reliability improvement. One example of a model is described in Brown, C., “Casting a wider, deeper net”, EE Times, Oct. 13, 2003.
3. Limitations of the Prior Art
Previous inventions have attempted to solve the collision avoidance problem for each vehicle independently of the other vehicles on the roadway. Systems that predict vehicle trajectories generally fail because two vehicles can be on a collision course and within the last 0.1 second, a slight change of direction avoids the collision. This is a common occurrence that depends on the actions of the individual drivers and no collision avoidance system now in existence is believed to be able to differentiate this case from an actual collision. In the present invention described below, every equipped vehicle will be confined to a corridor and to a position within that corridor where the corridor depends on sub-meter accurate digital maps. Only if that vehicle deviates from the corridor will an alarm sound or the vehicle control system take over control of the vehicle sufficiently to prevent the vehicle from leaving its corridor if an accident would result from the departure from that corridor.
Additionally, no prior art system is believed to have successfully used the GPS navigational system, or an augmented DGPS to locate a vehicle on a roadway with sufficient accuracy that that information can be used to prevent the equipped vehicle from leaving the roadway or striking another similarly equipped vehicle.
Prior art systems in addition to being poor at locating potential hazards on the roadway, have not been able to ascertain whether they are in fact on the roadway or off on the side, whether they are threatening vehicles, static signs, overpasses etc. In fact, no credible attempt to date has been made to identify or categorize objects which may impact the subject vehicle.
The RtZF™ system in accordance with this invention also contemplates a different kind of interrogating system. It is optionally based on scanning infrared laser radar, terahertz radar with or without range gating. This system, when used in conjunction with accurate maps, will permit a precise imaging of an object on the road in front of the vehicle, for example, permitting it to be identified (using neural networks) and its location, velocity and the probability of a collision to be determined.
In particular, the system of this invention is particularly effective in eliminating accidents at intersections caused by drivers running stop signs, red stoplights and turning into oncoming traffic. There are approximately one million such accidents and they are the largest killer of older drivers who frequently get confused at intersections.
4. Definitions
“Pattern recognition” as used herein will generally mean any system which processes a signal that is generated by an object (e.g., representative of a pattern of returned or received impulses, waves or other physical property specific to and/or characteristic of and/or representative of that object) or is modified by interacting with an object, in order to determine to which one of a set of classes that the object belongs. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally a series of electrical signals coming from transducers that are sensitive to acoustic (ultrasonic) or electromagnetic radiation (e.g., visible light, infrared radiation, capacitance or electric and/or magnetic fields), although other sources of information are frequently included. Pattern recognition systems generally involve the creation of a set of rules that permit the pattern to be recognized. These rules can be created by fuzzy logic systems, statistical correlations, or through sensor fusion methodologies as well as by trained pattern recognition systems such as neural networks, combination neural networks, cellular neural networks or support vector machines.
“Neural network” as used herein, unless stated otherwise, will generally mean a single neural network, a combination neural network, a cellular neural network, a support vector machine or any combinations thereof. For the purposes herein, a “neural network” is defined to include all such learning systems including cellular neural networks, support vector machines and other kernel-based learning systems and methods, cellular automata and all other pattern recognition methods and systems that learn. A “combination neural network” as used herein will generally apply to any combination of two or more neural networks as most broadly defined that are either connected together or that analyze all or a portion of the input data.
A “combination neural network” as used herein will generally apply to any combination of two or more neural networks that are either connected together or that analyze all or a portion of the input data. A combination neural network can be used to divide up tasks in solving a particular object sensing and identification problem. For example, one neural network can be used to identify an object occupying a space at the side of an automobile and a second neural network can be used to determine the position of the object or its location with respect to the vehicle, for example, in the blind spot. In another case, one neural network can be used merely to determine whether the data is similar to data upon which a main neural network has been trained or whether there is something significantly different about this data and therefore that the data should not be analyzed. Combination neural networks can sometimes be implemented as cellular neural networks. What has been described above is generally referred to as modular neural networks with and without feedback. Actually, the feedback does not have to be from the output to the input of the same neural network. The feedback from a downstream neural network could be input to an upstream neural network, for example. The neural networks can be combined in other ways, for example in a voting situation. Sometimes the data upon which the system is trained is sufficiently complex or imprecise that different views of the data will give different results. For example, a subset of transducers may be used to train one neural network and another subset to train a second neural network etc. The decision can then be based on a voting of the parallel neural networks, sometimes known as an ensemble neural network. In the past, neural networks have usually only been used in the form of a single neural network algorithm for identifying the occupancy state of the space near an automobile.
A trainable or a trained pattern recognition system as used herein generally means a pattern recognition system that is taught to recognize various patterns constituted within the signals by subjecting the system to a variety of examples. The most successful such system is the neural network used either singly or as a combination of neural networks. Thus, to generate the pattern recognition algorithm, test data is first obtained which constitutes a plurality of sets of returned waves, or wave patterns, or other information radiated or obtained from an object (or from the space in which the object will be situated in the passenger compartment, i.e., the space above the seat) and an indication of the identify of that object. A number of different objects are tested to obtain the unique patterns from each object. As such, the algorithm is generated, and stored in a computer processor, and which can later be applied to provide the identity of an object based on the wave pattern being received during use by a receiver connected to the processor and other information. For the purposes here, the identity of an object sometimes applies to not only the object itself but also to its location and/or orientation and velocity in the vicinity of the vehicle. For example, a vehicle that is stopped but pointing at the side of the host vehicle is different from the same vehicle that is approaching at such a velocity as to impact the host vehicle. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, sensor fusion, Kalman filters, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
A pattern recognition algorithm will thus generally mean an algorithm applying or obtained using any type of pattern recognition system, e.g., a neural network, sensor fusion, fuzzy logic, etc.
To “identify” as used herein will generally mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all motorcycles, one containing all trees, or all trees in the path of the host vehicle depending on the purpose of the system.
To “ascertain the identity of” as used herein with reference to an object will generally mean to determine the type or nature of the object (obtain information as to what the object is), i.e., that the object is an car, a car on a collision course with the host vehicle, a truck, a tree, a pedestrian, a deer etc.
A “rear seat” of a vehicle as used herein will generally mean any seat behind the front seat on which a driver sits. Thus, in minivans or other large vehicles where there are more than two rows of seats, each row of seats behind the driver is considered a rear seat and thus there may be more than one “rear seat” in such vehicles. The space behind the front seat includes any number of such rear seats as well as any trunk spaces or other rear areas such as are present in station wagons.
In the description herein on anticipatory sensing, the term “approaching” when used in connection with the mention of an object or vehicle approaching another will usually mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The “target” vehicle is the vehicle that is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
“Vehicle” as used herein includes any container that is movable either under its own power or using power from another vehicle. It includes, but is not limited to, automobiles, trucks, railroad cars, ships, airplanes, trailers, shipping containers, barges, etc. The word “container” will frequently be used interchangeably with vehicle however a container will generally mean that part of a vehicle that separate from and in some cases may exist separately and away from the source of motive power. Thus, a shipping container may exist in a shipping yard and a trailer may be parked in a parking lot without the tractor. The passenger compartment or a trunk of an automobile, on the other hand, are compartments of a container that generally only exists attaches to the vehicle chassis that also has an associated engine for moving the vehicle. Note a container can have one or a plurality of compartments.
“Transducer” or “transceiver” as used herein will generally mean the combination of a transmitter and a receiver. In come cases, the same device will serve both as the transmitter and receiver while in others two separate devices adjacent to each other will be used. In some cases, a transmitter is not used and in such cases transducer will mean only a receiver. Transducers include, for example, capacitive, inductive, ultrasonic, electromagnetic (antenna, CCD, CMOS arrays, laser, radar transmitter, terahertz transmitter and receiver, focal plane array, pin or avalanche diode, etc.), electric field, weight measuring or sensing devices. In some cases, a transducer will be a single pixel either acting alone, in a linear or an array of some other appropriate shape. In some cases, a transducer may comprise two parts such as the plates of a capacitor or the antennas of an electric field sensor. Sometimes, one antenna or plate will communicate with several other antennas or plates and thus for the purposes herein, a transducer will be broadly defined to refer, in most cases, to any one of the plates of a capacitor or antennas of a field sensor and in some other cases a pair of such plates or antennas will comprise a transducer as determined by the context in which the term is used.
A “wave sensor” or “wave transducer” is generally any device which senses either ultrasonic or electromagnetic waves. An electromagnetic wave sensor, for example, includes devices that sense any portion of the electromagnetic spectrum from ultraviolet down to a few hertz. The most commonly used kinds of electromagnetic wave sensors include CCD and CMOS arrays for sensing visible and/or infrared waves, millimeter wave and microwave radar, and capacitive or electric and/or magnetic field monitoring sensors that rely on the dielectric constant of the object occupying a space but also rely on the time variation of the field, expressed by waves as defined below, to determine a change in state.
A “CCD” will be defined to include all devices, including CMOS arrays, APS arrays, QWIP arrays or equivalent, artificial retinas and particularly HDRC arrays, which are capable of converting light frequencies, including infrared, visible and ultraviolet, into electrical signals. The particular CCD array used for many of the applications disclosed herein is implemented on a single chip that is less than two centimeters on a side. Data from the CCD array is digitized and sent serially to an electronic circuit (at times designated 120 herein) containing a microprocessor for analysis of the digitized data. In order to minimize the amount of data that needs to be stored, initial processing of the image data takes place as it is being received from the CCD array, as discussed in more detail above. In some cases, some image processing can take place on the chip such as described in the Kage et al. artificial retina article referenced above.
An “occupant protection apparatus” is any device, apparatus, system or component which is actuatable or deployable or includes a component which is actuatable or deployable for the purpose of attempting to reduce injury to the occupant in the event of a crash, rollover or other potential injurious event involving a vehicle
Inertial measurement unit (IMU), inertial navigation system (INS) and inertial reference unit (IRU) will in general be used be used interchangeably to mean a device having a plurality of accelerometers and a plurality of gyroscopes generally within the same package. Usually such a device will contain 3 accelerometers and 3 gyroscopes. In some cases a distinction will be made whereby the INS relates to an IMU or an IRU plus additional sensors and software such as a GPS, speedometer, odometer or other sensors plus optimizing software which may be based on a Kalman filter.
A precise positioning system or PPS is a system based on some information, usually of a physical nature, in the infrastructure that determines the precise location of a vehicle independently of a GPS-based system or the IMU. Such a system is employed as a vehicle is traveling and passes a particular location. A PPS can make use of various technologies including radar, laser radar, terahertz radar, RFID tags located in the infrastructure, MIR transmitters and receivers. Such locations are identified on a map database resident within the vehicle. In one case, for example, the map database contains data from a terahertz radar continuous scan of the environment to the side of a vehicle from a device located on a vehicle and pointed 45 degrees up relative to the horizontal plane. The map database contains the exact location of the vehicle that corresponds to the scan. Another vehicle can then determine its location by comparing its scan data with that stored with the map database and when there is a match, the vehicle knows its location. Of course many other technologies can be used to accomplish a similar result.
Unless stated otherwise, laser radar, lidar and ladar will be considered equivalent herein. In all cases they represent a projected laser beam, which can be in the visual part of the electromagnetic spectrum but generally will be the infrared part of the electromagnetic spectrum and usually in the near infrared wavelengths. The projected laser beam can emanate from the optics as a nearly parallel beam or as a beam that diverges at any desired angle from less than zero degrees to ten or more of degrees depending on the application. A particular implementation may use a laser beam that at one time diverges at an angle less than one degree and at another time may diverge at several degrees using adjustable optics. The laser beam can have a diameter as it leaves the vehicle ranging from less than a millimeter to several centimeters. The above represent typical or representative ranges of dimensions but this invention is not limited by these ranges.